In this lecture, we bridge the gap between clustering algorithms and the critical topic of model interpretability. We begin with a recap of DBSCAN and a demonstration of K-Means combined with PCA for visualization. The core focus then shifts to why interpretability matters, distinguishing between models that are inherently interpretable and those that require post-hoc explanation techniques.
Learning Objectives:
Understand the practical implementation of K-Means and PCA visualization
Articulate why interpretability is essential in high-stakes ML applications
Distinguish between Interpretable and Explainable ML
Compare Global vs. Local surrogate model approaches
Evaluate explainability methods using fidelity, accuracy, stability, and consistency
Theory
1. DBSCAN Recap
Density-Based Spatial Clustering of Applications with Noise (DBSCAN) groups together points that are closely packed together, marking points that lie alone in low-density regions as outliers. Unlike K-Means, DBSCAN does not require specifying the number of clusters beforehand and can discover clusters of arbitrary shape.
DBSCAN Parameters:
$$ \varepsilon \text{ (eps)} = \text{maximum distance between two samples for them to be neighbors} $$
$$ \text{MinPts} = \text{minimum number of points to form a dense region (core point)} $$
2. K-Means and PCA Visualization
When working with high-dimensional data, Principal Component Analysis (PCA) reduces dimensionality while preserving maximum variance. By projecting data onto the first two principal components, we can visualize K-Means clusters in 2D space.
PCA Objective: Find projection matrix $W$ that maximizes variance:
$$ \max_W \text{Tr}(W^T S W) \quad \text{subject to} \quad W^T W = I $$
where $S$ is the covariance matrix.
3. The Need for Interpretability
The need for interpretability arises from an incompleteness in problem formalization. For many real-world tasks, getting the prediction (the what) is not enough; we must also explain how the model arrived at that prediction (the why).
High-Stakes Example: A self-driving car's cyclist detector might achieve 99% accuracy in testing but fail dangerously if it learned to detect cyclists by recognizing bike lanes rather than the bicycles themselves. Without interpretability, we wouldn't discover this until accidents occur.
4. Detecting Bias
Machine learning models can pick up biases from training data, effectively turning them into discriminatory systems. Interpretability serves as a crucial debugging tool for detecting such bias.
Real-World Case: Amazon's Hiring Algorithm (Discontinued 2018)
Amazon developed an AI recruiting tool that penalized resumes containing the word "women's" (e.g., "women's chess club captain") because it learned from historical data where men dominated technical roles. The model accurately reflected past hiring patterns but perpetuated gender bias. Interpretability tools revealed this before widespread deployment.
5. Interpretable vs. Explainable ML
While often used interchangeably, a useful distinction exists:
Aspect
Interpretable ML
Explainable ML
Definition
Models transparent by design
Techniques explaining any model, including black boxes
Analogy
Glass box — you can see through it
Black box needing X-ray vision
Examples
Linear Regression, Decision Trees
LIME, SHAP for Neural Networks
Approach
Examine model structure itself
Use separate explanation methods
Figure: Spectrum of model interpretability from transparent to black-box models.
6. Taxonomy of Explainability Methods
Global Methods
Local Methods
Global Explainability
Explain the overall behavior of the model across the entire dataset.
Question Answered: "How does the model work in general?"
Examples: Global surrogate models, feature importance
Use Cases: Auditing for systematic bias, model documentation, regulatory compliance
Local Explainability
Explain individual predictions for specific instances.
Question Answered: "Why did the model make this particular prediction?"
Examples: LIME, SHAP for individual instances
Use Cases: Explaining loan rejections, debugging individual errors, personalized explanations
7. Global Surrogate Models
A global surrogate model is an interpretable model trained to approximate the predictions of a black-box model. We can draw conclusions about the black-box by interpreting the surrogate.
Goal: Approximate black-box function $f$ with surrogate $g$:
$$ g \approx f \quad \text{subject to} \quad g \text{ is interpretable} $$
Steps to Obtain a Global Surrogate:
Select dataset $X$ (training set or new data from same distribution)
Get predictions from the black-box model: $\hat{y}_{bb} = f(X)$
Select interpretable model type (linear model, decision tree, etc.)
Train interpretable model on $X$ and $\hat{y}_{bb}$
Measure how well surrogate replicates black-box predictions (fidelity)
Interpret the surrogate model
8. Evaluating Explainability Methods
Property
Definition
Measurement
Fidelity
How well does the explanation match the black box?
How correct are the explanations vs. ground truth?
Percentage correct on unseen data
Stability
Do similar inputs get similar explanations?
Variance of explanations for perturbed inputs
Consistency
Do different models give similar explanations?
Correlation between explanations
Key Insight: High fidelity does not imply high accuracy. A surrogate can perfectly mimic a bad model (high fidelity, low accuracy). Always check both metrics!
Interactive Examples
Interactive Fidelity Calculator
Enter black-box and surrogate predictions to compute fidelity metrics:
Scenario: Credit approval system with 5 test cases.
Case
Black Box Prediction (Prob)
Surrogate Prediction (Prob)
1
2
3
4
5
Interpretability Spectrum Explorer
Click on each model type to see its interpretability characteristics:
Linear Reg
Decision Tree
Random Forest
Neural Net
Hover over a model type above to see its interpretability profile.
Numerical Solutions
Problem: Computing Surrogate Model Fidelity
Scenario: A credit approval black-box model and its decision tree surrogate are tested on 1000 cases.
Model/Method
Correct Predictions (vs Ground Truth)
Match with Black Box
Black Box Model
900/1000 = 90% Accuracy
—
Explanation (Surrogate) Model
850/1000 = 85% Accuracy
950/1000 = 95% Fidelity
Step-by-Step Solution
Step 1: Understand the Metrics
Accuracy (85%): The surrogate is correct about the actual outcome 850 out of 1000 times.
Fidelity (95%): The surrogate agrees with the black box 950 out of 1000 times.
Step 2: Analyze the Gap
The 10% gap between fidelity and accuracy reveals two types of errors:
Faithfully replicated errors: 50 cases where both black box and surrogate are wrong (950 - 900 = 50)
New errors introduced by surrogate: 50 cases where surrogate disagrees with the correct black box prediction
Step 3: Interpret the Result
High fidelity (95%) means the surrogate is a trustworthy approximation of the black box. However, since the black box itself has 90% accuracy, the surrogate's 85% accuracy shows that simplifying the model introduces some additional error. This is the typical interpretability-accuracy trade-off.
$$ \text{Fidelity} = \frac{\text{Agreements between Surrogate and Black Box}}{\text{Total Cases}} = \frac{950}{1000} = 0.95 $$
A medical diagnosis black-box model achieves 88% accuracy on 500 test cases. A decision tree surrogate achieves 82% accuracy and 94% fidelity. How many cases show:
Both models agreeing on the correct diagnosis?
The black box correct but surrogate wrong?
Both models agreeing on the wrong diagnosis?
Solution:
Black box correct: 440 cases (88% of 500)
Surrogate correct: 410 cases (82% of 500)
Agreements (fidelity): 470 cases (94% of 500)
Disagreements: 30 cases
Let $x$ = both correct, $y$ = both wrong, $z$ = BB correct but surrogate wrong, $w$ = surrogate correct but BB wrong.
We know: $x + y = 470$ (fidelity), $x + z = 440$ (BB correct), $x + w = 410$ (surrogate correct), and $x + y + z + w = 500$.
Solving: $z + w = 30$. From $x + z = 440$ and $x + w = 410$, we get $z - w = 30$. Thus $z = 30, w = 0$.
Then $x = 410$ and $y = 60$.
Both correct: 410 cases
BB correct, surrogate wrong: 30 cases
Both wrong: 60 cases
Problem 2: Identifying the Right Patterns
A deep learning model for pneumonia detection achieves 96% accuracy on chest X-rays. An interpretability analysis reveals the model focuses primarily on hospital wristband tags in the image corners rather than lung patterns. Answer the following:
What type of interpretability issue is this?
Is the model's high accuracy trustworthy? Why or why not?
What should be done before deployment?
Solution:
This is a spurious correlation / shortcut learning issue. The model learned to associate wristband tags (which may correlate with patient severity or hospital protocols) with pneumonia rather than actual pathological features.
No, the high accuracy is not trustworthy for real-world deployment. The model will fail on images without wristbands or from different hospitals, and it does not actually understand pneumonia pathology.
Before deployment: retrain with wristband-removed/augmented data, use interpretability tools to verify attention on lung regions, and test on external datasets from different hospitals.
Problem 3: Global Surrogate Design
You have a random forest with 200 trees predicting house prices using 50 features. You want to build a global surrogate. Which model would you choose and what are the trade-offs of:
A decision tree with max_depth=2?
A linear regression with all 50 features?
A Lasso regression with 5 features?
Solution:
Decision Tree (depth=2): Very interpretable (2-3 rules), but likely low fidelity. Good for stakeholder communication, poor for capturing complex interactions.
Linear Regression (50 features): High potential fidelity but hard to interpret. Coefficients show direction and magnitude but with multicollinearity, interpretation becomes difficult.
Lasso (5 features): Good balance. Automatic feature selection gives interpretability while maintaining reasonable fidelity. Recommended approach for most use cases.
Interactive Quiz
Question 1: Which of the following is an example of an inherently interpretable model?
Decision Tree
Deep Neural Network
Convolutional Neural Network
Random Forest
Question 2: What does high fidelity of a surrogate model indicate?
The surrogate is more accurate than the black box
The surrogate closely mimics the black box's predictions
The black box has perfect accuracy
The model has no bias
Question 3: A global surrogate model is trained to:
Explain a single prediction instance
Approximate the overall behavior of a black-box model
Replace the original black-box model entirely
Reduce the dimensionality of the dataset
Question 4: In the Amazon hiring algorithm case, what was the primary issue?
The model had low accuracy on training data
The model learned gender bias from historical hiring data
The model was too simple to capture patterns
The model required too much computational power
Question 5: Which metric measures whether similar inputs receive similar explanations?
Fidelity
Consistency
Stability
Accuracy
Key Takeaways
Interpretability is essential in high-stakes applications where understanding the "why" behind predictions is as important as the predictions themselves.
Interpretable models (linear regression, decision trees) are transparent by design, while explainable ML techniques (LIME, SHAP) explain black-box models post-hoc.
Global surrogate models approximate the entire black-box model with an interpretable model, enabling overall behavior understanding.
Fidelity measures how well an explanation mimics the black box; accuracy measures correctness against ground truth. They are related but distinct.
High accuracy on historical data does not guarantee fairness or ethical behavior — interpretability tools are necessary for bias detection.
DBSCAN discovers clusters of arbitrary shape and handles noise, while K-Means + PCA provides powerful visualization capabilities for high-dimensional clustering.
Common Pitfalls
Assuming high fidelity means high accuracy: A surrogate can perfectly mimic a bad model. Always check the black box's accuracy independently.
Ignoring the interpretability-accuracy trade-off: Simpler surrogates are easier to interpret but may miss important nuances of complex models.
Using global explanations for local decisions: A global surrogate might hide important local variations. Different instances may need different explanations.
Deploying without bias checks: Models trained on biased historical data will perpetuate that bias unless explicitly audited using interpretability tools.
Confusing correlation with causation: Interpretability reveals what the model learned, not necessarily true causal relationships in the data.
Choosing inappropriate surrogate complexity: A decision tree with max_depth=1 is too simple; one with 50 levels is not interpretable. Aim for 3-5 levels.
Resources
📚 Interpreting Machine Learning Models by Nandi and Pal
📚 Interpretable Machine Learning by Christoph Molnar (available free online)